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Although afterschool programs for children have been operating for many years in some 
communities, the afterschool movement - the great national awakening to the opportunity 
afterschool offers - is just a few years old. As public demand for afterschool has grown, so has 
the demand for accountability. That is particularly true in afterschool programs that spend public 
dollars. After all, where tax dollars flow, so must accountability to taxpayers. 

The Landscape of Afterschool Evaluations 

A number of different types of evaluations have been conducted over the last several years, 
assessing various aspects of afterschool programming. Some evaluations seek to gather data on 
whether programs have been structured as they were originally intended, how well they have 
done at meeting attendance and staffing goals, how they “fit” in the school environment and 
more. Others explore the effect afterschool programs have on the children who participate in 
them, their parents, and even the communities at large. 

Both types of evaluations are of great value to afterschool providers and to policymakers, and 
when taken together the two types of studies help identify the particular program elements and 
approaches most critical to accomplishing program goals. It is useful, for example, to correlate 
information on student attendance at afterschool programs with student academic performance. 
Were an evaluator to conclude that attendance is key to academic gains, program designers 
might focus more energy on improving attendance for students. 

Evaluations also differ by virtue of who conducts them. Many programs self-evaluate, providing 
useful data and satisfying the needs of their various stakeholders - parents, funders, partnering 
businesses, local public officials and so on. But for academics and large funders - the federal 
government, state governments, the Open Society Institute, the Charles Stewart Mott Foundation 
or the Wallace Fund, for example - more exacting standards and greater independence is often 
required. Independent evaluations commissioned by such entities are the primary subject of this 
document. 

Moreover, this compilation focuses chiefly on the impact of afterschool programs on student 
academic achievement. A second backgrounder, available from the Afterschool Alliance 
website at www.afterschoolalliance.org, summarizes findings related to student safety, behavior, 
substance-abuse-prevention, and discipline. 
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Summary Lessons from the Data 

The data and conclusions from these studies amply demonstrate the positive effect afterschool 

programs have on student academic achievement. (Citations for the following are included in 

the detailed descriptions of afterschool studies that follow.) 

> Evaluations of LA’s BEST by the UCLA Center for the Study of Evaluation show that 
students’ regular school-day attendance improved once they began participating in the 
afterschool program. That improved attendance led to higher academic achievement on 
standardized tests of math, reading and language arts. In addition, language redesignation 
rates favored LA’s BEST students when compared with non-LA’s BEST students. 

> Children in the Ohio Urban School Initiative School Age Child Care Project (SACC) scored 
higher than non-participating students across the state, according to an evaluation by the 
University of Cincinnati College of Education’s Evaluation Services Center. “SACC 4th 
grade students’ scores exceeded the statewide percentages of students meeting proficiency 
standards in every subject area tested: writing, reading, mathematics, citizenship, and 
science. SACC 6th graders exceeded the statewide percentages of students meeting 
proficiency standards in four of the five areas: writing, reading, mathematics, and 
citizenship.” 

> An evaluation by the Department of Education, University of California at Irvine and 
Research Support Services, found that reading and mathematics gains of students in Los 
Angeles’ YS-CARE program, aimed at children from families on TANF, outpaced those of 
non-participating students, as measured by SAT-9 scores. 

> Policy Studies Associates’ second-year evaluation of The After-School Corporation’s 
(TASC’s) program “found significant differences in proficiency-level shifts among active 
participants and nonparticipants who scored in the lowest proficiency level on the 1998-99 
mathematics tests. In math, 3 1 percent of active participants scoring at the lowest proficiency 
level in 1998-99 scored at a higher proficiency level in 1999-2000, compared to 23 percent of 
nonparticipants who demonstrated the same improvement. Two percent of these active 
participants increased their perfonnance to grade level, compared to 1 percent of 
nonparticipants. A similar but less pronounced pattern was observed on the reading tests 
administered in grades 3-8. Among those scoring at the lowest proficiency level in 1998-99, 
45 percent of active participants improved their scores in 1999-2000 enough to move to a 
higher performance level, and 3 percent scored at grade level. Forty percent of 
nonparticipants who scored at the lowest proficiency level in 1998-99 increased their scores 
enough to move to a higher proficiency level a year later, and 2 percent reached grade level.” 

> Reading scores for San Diego’s “6 to 6” students improved, according to evaluator WestEd. 
Fifty-seven percent of students increased their reading scores over the course of the studied 
year, and SAT-9 reading scores increased. Nearly ten percent of children moved up into the 
25 th percentile or higher in 2000 by comparison to 1999 reading scores. 

> A five-site evaluation of the Boys & Girls Clubs’ national Project Leam program, found that 
“As program involvement increased, engagement in reading, use of verbal skills, writing, 
tutoring, and the study of geography all significantly (p<.05) increased as well.” Further, 
“[tjhere was also a direct and statistically significant (p<.05) relationship between program 
involvement and enjoyment of reading, use of verbal skills, writing, and geography.” 

Further, “[ajverage grade increases over the 30-month study period were greatest for 
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program youth. Program youth increased their average grades by 1 1 percent from baseline to 
the 30-month measurement while BGC comparison youth and non-BGC comparison youth, 
over the same period, increased their average grades by .4 percent and .3 percent, 
respectively.” 

> A statewide evaluation of California’s After School Education and Safety Program (formerly 
the After School Learning and Safe Neighborhoods Partnerships Program, ASLSNPP) by the 
University of California at Irvine demonstrated mathematics gains closely related to 
individual students’ level of participation in the program: “Among students who participated 
for more than 150 days (approximately 7.5 months), there was an increase of 4.9 percent in 
students above the 25 th percentile, an increase considerably larger than that found statewide. 
The scores suggest that, particularly for students who participate in the ASLSNPP for 
substantial periods of time, there is a closing of the gap in Math achievement between low- 
income and other students." [Emphasis in original.] 

> Forty-four percent of students in San Diego’s “6 to 6” program increased their SAT-9 math 
scores, according to an evaluation by WestED. 

> Policy Studies Associates’ study of TASC’s third year of operation concluded: “Students 
who participated in TASC after-school activities the most consistently and for the longest 
period of time experienced the greatest math gains, when compared to similar 
nonparticipants. Among students who participated actively in TASC projects in each year of 
their enrollment, students participating for two years gained an average of four scale-score 
points more on the city-wide standardized tests than similar nonparticipants. Among active 
participants, students participating for three years gained six points more than similar 
nonparticipants. Demonstrating the value of even higher levels of participation, students 
classified as ‘highly active’ (participating 80 percent or more of the days they were enrolled 
in the year and at least 80 days) gained six scale-score points more than similar 
nonparticipants after only two years of TASC participation.” 



Large-Scale Evaluations of Afterschool 

Among the many recently released studies of afterschool programs, two stand out for their scope: 
a meta-analysis conducted by Mid-Continent Research for Education and Learning that examines 
dozens of afterschool studies (some reported on separately in this backgrounder), and a study of 
selected 21 st Century Community Learning Centers programs conducted by Mathematica Policy 
Research. Both studies were funded by federal contracts. 

The Mid-continent Research for Education and Learning (McREL) Meta-Analysis 

In January 2004, Mid-continent Research for Education and Learning (McREL), released a meta- 
analysis of 53 separate studies of out-of-school time programs, after sifting through hundreds of 
studies spanning many years. The study is available at 

http://www.mcrel.org/PDF/SchoolImprovementReform/5032RR RSOSTeffectiveness.pdf, and 
is cited here as McREL Study. A meta-analysis is a statistical re-analysis of the results of several 
studies on a related topic, conducted for the purpose of integrating the studies’ findings, 
notwithstanding differences in their respective research methodologies. The studies selected for 
inclusion in the McREL meta-analysis all met rigorous methodological standards. The study was 
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conducted for the Institute of Education Sciences of the U.S. Department of Education on a 
federal contract. 

McREL’s findings affirmed the widely held view that afterschool programs help students 
achieve in school. McREL’s researchers summarized their findings: 

The synthesis resulted in statistically significant positive effects of OST [out of school 
time] on both reading and mathematics student achievement. The overall effect sizes 
ranged from .06 to . 13 for reading and from .09 to . 17 for mathematics, depending on the 
statistical model used for meta-analysis. Though numerically small, these results are 
important because they are based on strategies to supplement the regular school day and 
to prevent learning loss. Positive findings for supplementary programs that address the 
needs of low-achieving or at-risk students are therefore encouraging. Together, the 
results for reading and mathematics suggest that OST programs can significantly increase 
the achievement of these students by an average of one-tenth of a standard deviation 
compared to those students who do not participate in OST programs. . . . [McREL Study, 
page 2] 

In addition to the analyses of study outcomes, the syntheses of reading and mathematics 
studies described some common features among the studies in each content area. In 
reading, these were the links between student attendance and student achievement, the 
importance of staff quality, the development of academic and social skills, the 
implementation of a well-defined reading curriculum, and the prevention of learning loss. 
Common features highlighted in the mathematics studies were additional time for 
remediation, the use of tutoring, the use of counseling and mentoring, and the 
combination of recreation with mathematics instruction. Overall, the meta-analytic and 
narrative results lead to the following conclusions and implications for practice and 
policy related to OST and its evaluation: 

> OST strategies can have positive effects on the achievement of low-achieving or 
at-risk students in reading and mathematics. 

> The timeframes for delivering OST programs (i.e., after school or summer school) 
do not influence the effectiveness of OST strategies. 

> Students in early elementary grades are more likely than older elementary and 
middle school students to benefit from OST strategies for improving reading, 
while there are indications that the opposite is true for mathematics. 

> OST strategies need not focus solely on academic activities to have positive 
effects on student achievement. 

> Administrators of OST programs should monitor program implementation and 
student learning in order to determine the appropriate investment of time for 
specific OST strategies and activities. 

> OST strategies that provide one-on-one tutoring for low-achieving or at-risk 
students have strong positive effects on student achievement in reading. [McREL 
Study, pages 2-3] 
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The Mathematica Study 



As the McREL findings suggest, recent years have seen dozens of rigorous studies of afterschool 
programs. Although these studies’ specific findings vary depending on the programs examined, 
the evaluation measures applied, and the design of the studies, most reach the general conclusion 
that afterschool programs help keep children safe and help them succeed in school. In February 
2003, the first phase of a three-phase, federally funded study offered strikingly different 
conclusions, and immediately became the subject of great controversy. 

Conducted by Mathematica Policy Research, Inc. on a contract from the U.S. Department of 
Education and with supplemental funding from the Charles Stewart Mott Foundation, the study 
was originally intended as a tool for improving the quality of afterschool programming. The 
Bush Administration’s release of the first phase of the study just hours after submitting a budget 
calling for a 40-percent cut in 21 st Century Community Learning Centers funding, and the 
study’s subsequent use by the Administration as the sole justification for the cut, set the tone for 
its reception. In fact, Mathematica’s conclusions included no budget recommendations one way 
or the other. But the study did include a number of negative findings based on data-collection 
and evaluation methods that many in the research community deemed flawed, and that much of 
the afterschool community found lacking in credibility. 

In the study, Mathematica researchers separately examined elementary and middle school 21 st 
Century-funded programs, relying on data from the 2000-2001 school year, and using control 
and comparison groups. Among the controversial findings the authors and the Department of 
Education spotlighted was that the first year of the evaluation found no statistically significant 
across-the-board academic gains, even though certain subgroups of students did show gains. 

Other more positive findings received less attention from the authors and the Administration, and 
consequently from the media: 

• African American and Hispanic students participating in afterschool programs showed 
significant academic gains. African American students in programs had higher scores on 
standardized math and reading tests, a reduced incidence of being absent or tardy for school, 
and were judged to have shown increased effort in the classroom. Hispanic students had 
higher math scores and reduced absence and tardiness. 

• Girls in 21 st CCLC programs showed significant gains in mathematics and in class 
participation - two areas that have long worried educators. 

• Student participation in afterschool produced greater involvement by parents of participating 
students - improving parents’ participation rates generally and by as much as 40 percent in 
some areas. Parental involvement has long been regarded as one of the keys to improved 
academic performance, and afterschool programs around the nation typically work to bring 
parents and other adults into the school building during the afterschool hours. Advocates say 
the Mathematica study’s finding regarding increased parental involvement, even at this early 
stage of the research, suggests that this program design is yielding important benefits. 

• Many trained, experienced teachers work in afterschool, bringing their skills and expertise to 
these programs. One-third of the program coordinators and three in five program staff 
members at programs in the study were school-day teachers. The middle school teachers who 
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worked in afterschool programs noted that, as a result of working with students at the 
afterschool learning centers, they improved their teaching skills and had better relationships 
with some students - another significant benefit of the program design. 

Controversy over the study was not solely the product of the startlingly negative conclusions 

highlighted by researchers. Their methodology also came under fire from fellow researchers, 

including several who served on an advisory committee to Mathematica on the study. Among 

the methodological and conceptual problems: 

• The emphasis of the 21 st CCLC program changed after the study began — becoming more 
tightly focused on academic achievement. In its reporting requirements for grantees, the 
Department of Education did not begin requiring data on student grades and achievement-test 
scores until the 2000-2001 school year. As a result, many programs did not particularly 
emphasize academic achievement. In fact, three in ten of the studied middle-school programs 
described improving academic performance as a “minor objective” of the program, choosing 
instead to focus on recreational, social or cultural development — a choice consistent with the 
21 st CCLC program’s guidelines at the time. 

• Just seven elementary school 21 st CCLC grantees participated in the evaluation - far fewer 
than one might expect for a study of this size. In an acknowledgement of this problem, 
Mathematica added more elementary school sites to the sample for future phases of the 
research, and conceded that “the elementary school findings in this report should be viewed as 
preliminary.” 

• Another problem with the sample of elementary schools, according to Mathematica: “grantees 
were chosen for their ability to carry out the experimental design” of the evaluation, rather 
than because they were representative of the population of the 21 st Century program. 

Therefore, the researchers conclude, “L hidings for the elementary school centers in the 
evaluation do not generalize to all elementary school centers.” 

• The samples of Hispanic and African American students in the studied elementary schools are 
not representative of the number of Hispanic and African American students in afterschool 
programs. According to Mathematica’s own numbers, 27.6 percent of elementary afterschool 
students in 21 st CCLC programs were Hispanic, compared to only 1.8 percent of the students 
in the studied elementary programs. Similarly, 22.8 percent were African American, while 
66.8 percent of students in the studied programs were African American. 

• White students were over-represented in the student populations of the 34 participating middle 
school grantees. By comparison to 21 st CCLC middle school centers across the nation, the 
studied centers included more white children, fewer African American, fewer Native 
American or Pacific Islander, fewer American Indian or Alaska Native, and fewer Asian. 

• Beyond the demographics, students in the middle school sample were different in ways that 
might affect academic achievement. Before their participation in the afterschool program year 
studied by Mathematica, the middle school students were, as a group, less likely than students 
in the comparison sample to do assigned homework, less likely to read for fun, more likely to 
watch television, less confident in their reading skills, more likely to expect to drop out of high 
school, less likely to have a parent with a college degree and more likely to have a parent who 
dropped out of high school. 
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Many scholars and afterschool leaders have also questioned the Administration’s use of the study 
as its sole measure of the value of afterschool programs, noting the importance of reasonable 
expectations, and citing test-score and grade improvements as inappropriate early measures of 
afterschool programs’ value. As Robert Granger, president of the William T. Grant Foundation, 
and Thomas Kane, professor of policy studies and economics at UCLA wrote in a recent 
overview of afterschool evaluations [“Improving the Quality of After-School Programs,” 
Education Week, February 18, 2004, Robert C. Granger & Thomas Kane]: 

We need to be more realistic about what it takes to create discernible effects on 
achievement-test scores. In the national samples used to nonn the Stanford 
Achievement Test-9th Edition, 5th grade students scored only one-third of a 
standard deviation higher than 4th graders on reading, and one-half of a standard 
deviation higher on math. This reading-score difference is about as large as the 
difference in moving from 1000 to 1070 points on the combined SAT, or 100 to 
105 on an IQ test. In other words, everything that happens to a student between 
the end of 4th grade and the end of 5th grade — a whole school year of full-day 
classroom instruction, interactions with family, conversations with friends, and 
homework - is associated with an important but not huge gain on an achievement 
test. 

With this as a backdrop, consider the typical after-school program, with youths 
attending one to two days per week for two to three hours per day. While it is 
reasonable to expect that after-school activities can affect performance as 
measured by achievement tests, it is likely that such effects will be small. This is 
particularly true for reading scores, since they are traditionally less responsive 
than mathematics scores to instruction. 

Therefore, even if the programs are helping, effects on achievement tests are 
likely to be hard to detect statistically. We should balance a focus on test scores 
with an examination of intennediate effects - more parental involvement in 
school-related activities, more diligent homework completion, more school 
attendance, and better grades, for example - which may pay off in improved test 
performance over time. 

The second phase of Mathematica’s study is expected sometime in 2004, although the report’s 
timing has not yet been announced. It will include, according to Mathematica, a second year of 
data for middle school programs, and additional first-year data from elementary school programs 
not included in the first report. Of particular interest to the afterschool community will be the 
extent to which Mathematica and the Administration address the first phase’s methodological 
flaws and more fairly balance the emphasis they give to positive and negative conclusions. A 
third phase of the report will include a second year of elementary school data. 

Afterschool Evaluations in Detail 

The gulf between Mathematica’s findings and the large body of afterschool evaluation data 
before and since could not be more pronounced. Over the past decade a number of important 
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afterschool evaluations have been conducted - more than enough to demonstrate that afterschool 
programs help children achieve academically. Following are summaries of several of the most 
extensive evaluations. 

LA’s BEST 

Los Angeles’s Better Educated Students for Tomorrow, or LA’s BEST, is among the largest and 
best known afterschool programs in the nation. Launched in 1988 as a partnership between the 
Los Angeles Unified School District (LAUSD), the city of Los Angeles and the private sector, 
the program operates at 1 17 school sites, serving more than 19,000 students. Schools are chosen 
for participation because of the generally low academic achievement among their students, or 
because of the low economic status of the community, or high gang or crime rates in the 
neighborhood. 

Since early in the life of the program, the UCLA Center for the Study of Evaluation has 
conducted evaluation studies. The studies have focused on a variety of topics, using a range of 
measures. The Center released its separate studies in March 1990, March 1991, July 1991, 
December 1993 and spring 1995. Then in June 2000, the Center released a comprehensive 
report summarizing each of the five previous studies and adding a rich set of findings based on 
its five-year tracking of the academic performance and school attendance of LA’s BEST students 
who were in 2 nd through 5 th grades in the 1993-94 school year. 

The study’s description of its methodology: “To study LA’s BEST schools, we obtained 
information about students including ethnicity, gender, language proficiency status, eligibility for 
free/reduced lunch (the proxy for low-income level) and disability status. In addition, we 
collected outcome data including achievement test scores (using either the Comprehensive Test 
of Basic Skills [CTBS] or the more recently adopted Stanford-9 Achievement Test [SAT-9] in 
reading, mathematics, and language arts. The rate at which students were redesignated as fully 
proficient in English was also collected. We also obtained school absence rates, course-taking 
patterns and rates of student mobility (moving between schools or out of the district).” 

In addition, because of the size of the LA’s BEST program and of the LAUSD school system, 
researchers were able to track an extraordinarily large sample of students and a correspondingly 
large “control” group - more than 4,000 LA’s BEST students and more than 15,000 non- 
participating students. The sheer numbers of students tracked make the data produced highly 
reliable. 

The findings, summarized at http://www.lasbest.org/learn/eval.html with a link to the complete 
study, are powerful evidence of the value of afterschool programming. In short, the study found 
that LA’s BEST participants, defined as students who participated regularly and over a period of 
more than one year, when compared to non-participating students, were absent less from school, 
“show positive achievement on standardized tests in mathematics, reading and language arts,” 
and had “higher language redesignation rates to English proficiency.” [Quoting from LA’s 
BEST’S summary of the findings, at http://www.lasbest.org/learn/eval.html .] Specific findings: 

> “[0]ur results show that higher levels of participation in LA’s BEST led to better subsequent 
school attendance, which in turn related to higher academic achievement on standardized 
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tests of mathematics, reading and language arts.” [A Decade of Results: The Impact of the 
LA ’.s' BEST After School Enrichment Program on Subsequent Studen t Achievement and 
Performance, a longitudinal study report and a synthesis of research begun in 1990 by the 
UCLA Center for the Study of Evaluation with support from the Bandai Foundation and the 
City of Los Angeles. June 2000. Denise Huang, Barry Gribbons, Kyung Sung Kim, 
Charlotte Lee, Eva L. Baker. Page 7.] 

> “Language redesignation rates favored LA’s BEST students when compared with non-LA’s 
BEST students for the cohort analyzed (fourth grade, 1994-1995). Significant differences in 
favor of LA’s BEST students were found for subsequent redesignation rates in Grades 6 and 
8. No significant differences were found in comparing performance for Grades 5 and 7.” 
[Decade of Results at page 8.] 

> “Absence follow-up data for the fifth-grade cohort (1994-1995) showed that students who 
participated in LA’s BEST had significantly fewer absences in Grades 6 and 7, although no 
differences were detected in Grades 8 and 9.” [ Decade of Results at page 8.] 

> “Although in the initial year LA’s BEST students began with statistically significant 
mathematics achievement scores lower than those of non-participants, in 1997-1998 those 
differences no longer existed.” [ Decade of Results at page 9.] 

Researchers conclude: “From our perspective, it looks as if LA’s BEST is a program that, when 
followed as a regular part of students’ broad educational experience, results in statistically 
important differences in student outcomes. The fact that we can detect any change on 
standardized achievement measures in itself is notable, for most educational interventions are 
unable to show impact on measures not tightly tied to the curriculum, or on follow-up 
achievement after a particular program is over. On a practical level, LA’s BEST needs to focus 
its attention on increasing the attendance of enrolled students. It may be that high-level attenders 
do so because they and their parents are more highly motivated, and this interest transfers to 
achievement. But it is equally likely that coming to school and to the LA’s BEST program 
regularly is the reason for good performance and persisting impact subsequent to leaving LA’s 
BEST.” [ Decade of Results at pages 9-10.] 

After School Education and Safety Program - California 

Begun in 1998 as the California Afterschool Learning and Safe Neighborhoods Partnerships 
Program, the After School Education and Safety Program (ASESP) provides $117 million 
annually in matching funds to local partnerships of school districts, community groups, and local 
governments to provide before and afterschool programs for students. In selecting grantees, the 
state gives priority to schools where 50 percent or more of pupils are eligible for free or reduced- 
cost meals. Programs are designed locally, but are required to include an “educational and 
literacy component to provide tutoring or homework assistance in one or more of the following 
subject areas: language arts, mathematics, history and social science, or science”; and “an 
educational enrichment component, which may include but is not limited to, recreation and 
prevention activities. Such activities might involve the arts, music, physical activity, health 
promotion, and general recreation; work preparation activities; community service-learning; and 
other youth development activities based on student needs and interests.” [ California ’s Before 
and After School Learning and Safe Neighborhoods Partnerships Program Fact Sheet, at 
http://www.cde.ca.gov/afterschool/aspfactsheetapr02.doc, September 3, 2002.] In all, 947 
afterschool programs were funded during the ASESP ’s first two years. 
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Working with the California Department of Education, the Education Department of the 
University of California at Irvine conducted evaluations of two academic years of the program, 
from 1999 to 2001, releasing results in February 2002. The evaluation relied on data supplied to 
the state by participating programs, as required by law, and examined student and parent 
satisfaction with their programs, as well as students’ academic outcomes. 

Findings included: 

> SAT-9 scores of participating students increased faster than those of students statewide. In 
reading, 4.2 percent of afterschool students moved from out of the lowest 25 percent of their 
classes. “This increase is more than twice the increase found among all students statewide 
(1.9 percent). . [ Evaluation of California ’s After School Learning and Safe Neighborhoods 
Partnerships Program, Department of Education, University of California at Irvine, February 
1, 2002, page 4.] In math, similar findings: 2.5 percent of afterschool participants moved 
out of the lowest quartile, compared with 1.9 percent statewide. [. Evaluation , page 6.] 

> Significantly, gains were closely related to individual students’ levels of participation in the 
program. “Among students who participated for more than 150 days (approximately 7.5 
months), there was an increase of 4.9 percent in students above the 25 th percentile, an 
increase considerably larger than that found statewide. The scores suggest that, particularly 
for students who participate in the ASLSNPP for substantial periods of time, there is a 
closing of the gap in Math achievement between low-income and other students .” 

[Evaluation, page 6. Emphasis in original.] 

> “The regular school day attendance of students in the ASLSNPP increased between 1999 and 
2000. Among the ASLSNPP participants who were absent 5 percent or more days in 1999, 
the average increase in attendance was 5.6 days. Among those who were absent 10 percent 
or more days in 1999, the average increase in attendance was 1 1 days. For those absent 15 
percent or more days in 1999, the average increase in attendance was 17 days.” [Harvard 
Family Research Project, Summary of ASLSNPP Evaluation, 

http://www.gse.harvard.edu/~hfrp/proiects/afterschool/mott/aslsnpp.pdf, September 3, 

2002 . ]‘ 

The After-School Corporation 

The After-School Corporation (TASC) is a New York City-based nonprofit, established by the 
Open Society Institute in 1998, representing a $25 million five-year commitment by the 
foundation. TASC provides grants to nonprofit organizations to establish partnerships with 
individual public schools, and the resulting afterschool programs follow a core set of program 
components. In all, 143 public schools in New York City and 73 schools in other parts of New 
York State participate. Funding is based on enrollment and is $1,000 per student, excluding 
start-up, facilities and staff training. 



1 The Harvard Family Research Project (HFRP) maintains a rich database of information on out-of-school-time 
evaluations, from which much information in this report is drawn. A complete listing of HFRP’s summaries is 
available at http://www.gse.harvard.edu/hfrp/projects/afterschool/evaldatabase.html. 
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The Charles Stewart Mott Foundation and the Carnegie Corporation, joined by the W.T. Grant 
Foundation and the Atlantic Philanthropies provided funding to the Washington-based Policy 
Studies Associates to conduct a five-year evaluation, including annual summary reports. The 
first year’s evaluation, covering the 1998-99 school year, focused largely on issues related to 
program design and participation. The second- and third-year evaluations focused more on 
academic achievement. The second-year evaluation, covering the 1999-2000 school year, found: 

> “Students reported feeling safe, relaxed, happy, and connected to their after-school program. 
When asked what they especially liked, they mentioned being with their friends, completing 
their homework before going home, and participating in activities that differed from those of 
the regular school day.” [ Building Quality and Supporting Expansion of After-school 
Projects, Summary of Findings, http://www.tascorp.org/pages/promising es2.pdf, page 12.] 

> The evaluation “found significant differences in proficiency-level shifts among active 
participants and nonparticipants who scored in the lowest proficiency level on the 1998-99 
mathematics tests. In math, 3 1 percent of active participants scoring at the lowest proficiency 
level in 1998-99 scored at a higher proficiency level in 1999-2000, compared to 23 percent of 
nonparticipants who demonstrated the same improvement. Two percent of these active 
participants increased their performance to grade level, compared to 1 percent of 
nonparticipants. A similar but less pronounced pattern was observed on the reading tests 
administered in grades 3-8. Among those scoring at the lowest proficiency level in 1998-99, 
45 percent of active participants improved their scores in 1999-2000 enough to move to a 
higher performance level, and 3 percent scored at grade level. Forty percent of 
nonparticipants who scored at the lowest proficiency level in 1998-99 increased their scores 
enough to move to a higher proficiency level a year later, and 2 percent reached grade level.” 
[Patterns of Student-Level Change Linked to TASC Participation Based on TASC Projects in 
Year 2, Executive Summary, 

http://www.policvstudies.com/studies/youth/Y2%20Performance%20Executive%20Summar 
V-Pdf , page 4.] 

> “Forty-five percent of principals in Year 2 reported that the TASC project has increased 
parents’ attendance at school events and 36 percent said that the project had increased 
parents’ attendance at parent-teacher conferences.” Ninety-seven percent of parents 
surveyed indicated that “their child liked to come to the program”; 86 percent agreed “that 
the project was helping their child academically.” Parents also said that the program helped 
them balance work and family life: 94 percent said the program was convenient; 60 percent 
said they missed less work than before because of the program; 59 percent said it supported 
them in keeping their job; and 54 percent said it was supportive to them in allowing them to 
work more hours. [Building Quality, page 15.] 

The third year of the evaluation, covering the 2000-2001 school year, concluded: 

> “Students who were active participants in TASC projects for more than a year showed 
significantly greater gains on citywide math tests than did similar nonparticipating 
classmates. Students who participated in TASC after-school activities the most consistently 
and for the longest period of time experienced the greatest math gains, when compared to 
similar nonparticipants. Among students who participated actively in TASC projects in each 
year of their enrollment, students participating for two years gained an average of four scale - 
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score points more on the city-wide standardized tests than similar nonparticipants. Among 
active participants, students participating for three years gained six points more than similar 
nonparticipants. Demonstrating the value of even higher levels of participation, students 
classified as ‘highly active’ (participating 80 percent or more of the days they were enrolled 
in the year and at least 80 days) gained six scale-score points more than similar 
nonparticipants after only two years of TASC participation. The performance of TASC 
participants on the citywide tests of reading and English/language arts was not significantly 
different from that of similar nonparticipants.” [What Have We Learned from TASC’s First 
Three Years? Evaluation of the TASC After-School Program, December 2002, page 7, at 
http://www.tascorp.org/pages/psaYear3.pdf 1 

> “In general, the TASC participants who were at greatest academic risk made the largest math 
gains, when compared to other students. (Reporting of subgroup analyses focuses here on 
math because of the consistent relationships with TASC participation, as found in the 
aggregate analyses of math achievement.) Math benefits were clearly evident for students 
who scored in the lowest of four proficiency levels in the year prior to TASC participation. 
The gains for these low-achieving students were evident for active participants regardless of 
their number of years of participation. Among students from low-income families, the 
evaluation also found evidence of after-school benefits in math after two or more years of 
active participation.” [What Have We Learned, page 7.] 

> “Among the various subgroups examined, African-American students were especially likely 
to benefit from active participation in TASC projects, demonstrating gains in math over 
similar nonparticipants after one or more years of active participation. Hispanic students 
benefited in math after two years of participation.” [What Have We Learned, page 7.] 

The final installment of the Policy Studies evaluation of TASC covered data from four full years 
of program operations. The final report was not available as this document went to press, but a 
four-page summary available from Policy Studies Associates summarized major findings of the 
report to come. When all was said and done, researchers concluded: 

> School principals reported significant benefits for students who participated in TASC 
programs. Ninety-five percent said that TASC gave students access to activities not available 
during the regular school day, 79 percent said that participants’ parents expressed more 
positive feelings about the school than before, and 66 percent said that after-school 
participants received special opportunities to hone literacy skills. [This and subsequent 
quotations from this study are from Building Quality, Scale, and Effectiveness in After- 
School Programs: Major Findings of the TASC Evaluation, Policy Studies Associates, Inc., 
page 3.] 

> Parents also expressed satisfaction with TASC services, based, as reported in surveys 
administered to a convenience sample of parents, on their children’s enjoyment of program 
activities, their children’s completion of homework before coming home, and the reassurance 
of knowing that their children were safe and cared for after school, which allowed parents to 
work more hours and miss work less often. 

> Analyses of data on academic performance and school attendance show that participation in 
TASC activities was linked to improvements in both areas, especially for students who 
participated regularly in TASC programming over two consecutive years. At the elementary- 
and middle-grades level: 
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> Analyses indicate after-school benefits in mathematics achievement, especially for those 
participants who attended TASC programs regularly and for more than a year.... Comparable 
findings in reading and English language arts were not apparent, although participants in 
some programs consistently out-gained comparable nonparticipants in this area. 

> Participants showed greater gains in school attendance than did nonparticipants, with the 
greatest benefits at grades 5-8. 

At the high school level: 

> After-school participants passed more Regents exams and earned more high school credits 
than nonparticipants, but conclusions from this evidence are limited by the fact that high 
school participants were already achieving at substantially higher levels than nonparticipants 
before entering a TASC project. 

> The difference that reflects the clearest association with TASC high school participation is in 
improved school attendance. When compared to nonparticipants with similar records of prior 
school attendance, TASC participants showed significantly more positive school attendance 
after a year of TASC participation (although the attendance of both participants and 
nonparticipants declined over grades 9-12). For example, the difference in the attendance 
gain of participants and similar nonparticipants in the lowest attendance quartile during the 
year before enrolling in TASC was +2.4 percentage points over one year, or the equivalent of 
a net gain of +4.4 school days in a 181-day school year (effect size of +0.27). 



The Boys & Girls Clubs’ Project Learn 

Begun in 1996, the Boys & Girls Clubs’ Project Learn focuses on providing youngsters with 
“high-yield learning activities,” including weekly discussions with knowledgeable adults, leisure 
reading, writing activities, homework help, helping others, and games that rely on cognitive 
skills. The program has been implemented in full at one-tenth of the Clubs’ 3,300 sites, and all 
sites are implementing components of the program. Steven P. Schinke, Ph.D., of New York’s 
Columbia School of Social Work led an evaluation of the program that relied on a quasi- 
experimental design. Three groups of students were identified, in five separate cities. The 
groups: 



• Students participating in Boys & Girls Clubs programs that had implemented Project 
Learn, (BGC program sites), 

• In the same cities, students at Boys & Girls Clubs that had not implemented Project Learn 
(BGC comparison sites), and 

• In the same cities, students at non-Boys & Girls Club sites that had not implemented the 
kind of enhanced learning initiatives characteristic of Project Leam (non-BGC 
comparison sites). 

All students in all groups lived in public housing projects, and the sites were chosen to be 
nationally representative of students in public housing. Data on students’ academic performance 
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were collected four times: before they began the program, six months after they began, 18 

months after they began, and 30 months after they began. Findings included: 

> “The level of program involvement, as rated by teachers on a scale of 0 to 10, was found to 
be associated with a number of self-reported academic outcomes.” [A Profile of the 
Evaluation of the Boys & Girls Clubs of America — Project Learn/Educational Enhancement 
Program, Harvard Family Research Project, 

http://www.gse.harvard.edu/hfrp/proiects/afterschool/mott/pleep.html . Based on Enhancing 
the Educational Achievement of At-Risk Youth, Steven P. Schinke, Ph.D. 

> “As program involvement increased, engagement in reading, use of verbal skills, writing, 
tutoring, and the study of geography all significantly (p<.05) increased as well.” 

> “There was also a direct and statistically significant (p<.05) relationship between program 
involvement and enjoyment of reading, use of verbal skills, writing, and geography.” 

> “At final follow-up (30 months after the program began), program youth more than BGC 
comparison youth and comparison youth more than non-BGC youth reported greater 
engagement in reading, enjoyment of reading, engagement in verbal activities, enjoyment of 
verbal activities, engagement in writing, enjoyment of writing, engagement in tutoring, 
enjoyment of tutoring, and enjoyment of geography.” 

> “Also at 30-month data collection, relative to the non-BGC comparison group youth, 
program and BGC comparison youth reported greater study of geography, more engagement 
in board games, enjoyment of board games, engagement in life-enhancement activities, and 
enjoyment of life-enhancement activities.” 

> “Data from teacher reports at final follow-up reveal that program and BGC comparison youth 
more than non-BGC comparison youth had more positive reading skills, writing skills, games 
skills, overall school performance, and interest in class material.” 

> “School grades at 30-month follow-up favored program youth over BGC comparison youth 
and non-BGC comparison youth on overall averages, reading scores, spelling scores, history 
scores, science scores, social studies scores, and attendance. At the 30-month follow-up, 
program and BGC comparison youth had better grades in math than non-BGC comparison 
youth.” 

> “Average grade increases over the 30-month study period were greatest for program youth. 
Program youth increased their average grades by 1 1 percent from baseline to the 30-month 
measurement while BGC comparison youth and non-BGC comparison youth, over the same 
period, increased their average grades by .4 percent and .3 percent, respectively.” 

> “Program youth missed an average of only 2.19 days of school a year at the 30-month 
measurement as compared to missing an average of 6.4 days a year at baseline. In contrast, 
BGC comparison youth went from missing an average of 4.85 days of school in the baseline 
year to missing an average of 12.33 days a year at the 30-month follow-up. Similarly, non- 
BGC youth went from 7.47 days at baseline to 16.67 at follow-up. The differences between 
the school attendance of program youth and youth in both comparison groups at 30 months 
were statistically significant at p<.05, while they had not been significantly different at 
baseline.” [All bullets from HFRP Profile.] 

Houston ’s After-School Achievement Program (ASAP) 

In 1997, Houston’s After-School Achievement Program began providing significant funding for 

afterschool programs in the city. The program has grown steadily since, and in the 2000-2001 
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academic year, it provided $2.3 million to 95 sites. ASAP has six programmatic goals: to 
reduce crime committed by and against juveniles; to prevent delinquency; to provide a safe, 
supervised place for youth; to provide academic enhancement and enrichment; to promote school 
attendance and discourage school drop-out; and to motivate youth to develop good citizenship. 

The program has been evaluated annually by independent evaluators, with the most recent study 
conducted by Dennis W. Smith, Ph.D. and James J. Zhang, P.E.D, covering the 2000-2001 
school year. Among their findings: 

> “In both science and fine arts, ASAP participants improved significantly over the course of 
the school year, compared with students not in the ASAP. While student achievement in the 
remaining nine subject areas [reading, other language arts, mathematics, social studies, 
handwriting, physical education, health and safety, computers and science lab] was not 
significantly different between the ASAP and non- ASAP groups, the post-test mean scores 
for ASAP participants clearly indicated improvement over the span of the program year.” 
[Shaping our Children ’s Future: Keeping a Promise in Houston Communities, 2001, Year 4 
Evaluation of the After-School Achievement Program, page 4, published by ASAP. Contact: 
Jennifer Brimer, ASAP Coordinator, 713-437-6981.] 

> Surveys conducted for the evaluation found that 22 percent of parents of ASAP children said 
“their children would be by themselves” without ASAP, and “close to 16 percent of parents 
said that their children would be watched by a sibling.” [Shaping, page 6.] 

San Diego’s ‘ 6 to 6’ Extended School Day Program 

San Diego has developed one of the nation’s most ambitious afterschool programs, with the goal 
of making affordable programs before and after school available to every elementary and middle 
school student in the City of San Diego. Two significant evaluations of the program have been 
conducted, one an interim report by WestED, released in April 2001, the other by Hoffman Clark 
and Associates released in July 2001. Using random sampling of sites, document review, 
interviews, focus groups and site observations, WestED found: 

> Parents expressed high levels of satisfaction with the program, “including their perceptions 
of the quality of academic enrichment, the degree to which children looked forward to the 
program, communication with staff, success at helping children complete homework, and the 
promotion of positive behavior in children.” [Harvard Family Research Project website at 
http://www.gse.harvard.edu/~hfrp/proiects/afterschool/mott/sd66esdp.pdf, hereafter HFRP- 
SD.] 

> “Almost two-thirds of responding parents noticed improvements in their children’s academic 
performance.” [HFRP-SD.] 

Relying on random sampling of program participants and reviewing a variety of data, WestED’s 
study concluded: 

> Reading scores for “6 to 6” students improved. Fifty-seven percent of students increased 
their reading scores over the course of the studied year, and SAT-9 reading scores increased. 
[HFRP-SD.] 
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> Nearly ten percent of children moved up into the 25 th percentile or higher in 2000 by 
comparison to 1999 reading scores. [HFRP-SD.] 

> Forty-four percent of students increased their SAT-9 math scores. [HFRP-SD.] 

Ohio Urban School Initiative School Age Child Care Project 

The Urban School Initiative School Age Child Care Project (SACC) funds a variety of 
afterschool programs in Ohio urban school districts. The University of Cincinnati College of 
Education’s Evaluation Services Center conducted a thorough review of the program’s 1998- 
1999 school year, measuring both project design and its outcomes. Data collection included 
document reviews, observation of programs, surveys and questionnaires. Among the findings: 

> “Ohio Proficiency Tests scores for both 4th and 6th graders showed that SACC children 
exceeded the state- wide percentages of students meeting proficiency standards. SACC 4th 
grade students’ scores exceeded the statewide percentages of students meeting proficiency 
standards in every subject area tested: writing, reading mathematics, citizenship, and science. 
SACC 6th graders exceeded the statewide percentages of students meeting proficiency 
standards in four of the five areas: writing, reading mathematics, and citizenship.” [Harvard 
Family Research Project at 

http://www.gse.harvard.edu/~hfrp/proiects/afterschool/mott/osisaccp.pdf , hereafter HFRP- 
Ohio.] 

> “School absence and tardiness were reduced for participating students. First graders who 
were not in a SACC program during kindergarten reduced the number of school days they 
missed from an average of 8 during their kindergarten year to an average of 3 days during 
their 1998-99 1st grade year. Eighth graders who were not in a SACC program during 7th 
grade reduced the average number of school days missed from 18 to 5.” [HFRP-Ohio.] 

> “Parents participating in interviews or completing surveys felt the programs had positive 
impacts on their families.” [HFRP-Ohio.] 

Pathways to Progress, St. Paul, Minnesota Public Schools 

Operating with a three-year 2 1 st Century Community Learning Centers grant, in 2000 the St. 
Paul, Minnesota public schools established Pathways to Progress, an eight-site afterschool 
program. Between 2000 and 2003, the sites served more than 3,000 students, one-third of all 
students enrolled at the eight participating schools. 

Researchers Kyla Wahlstrom, Tim Sheldon, and Ashley Murphy of the Center for Applied 
Research and Educational Improvement of the University of Minnesota 
(http://education.umn.edu/CAREI/default.html) conducted an evaluation of the program, 
assessing its success in achieving its principal objectives, among them, increasing student 
academic achievement. Released in March 2004, the evaluation used a matched-pair technique, 
comparing students who had attended the program for 30 or more days each year with matched 
students who had not participated in the program. It found that: 

> “In reading and math, more Pathways participants scored above the national norm on the 
SAT 10 and fewer students scored in the lowest quartile.” 

> “Data gathered on both two-year and three-year regular participants indicate the groups 
experienced continuous progress in their standardized test scores for both reading and 
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math. . . . Regular participants frequently began with lower scores in the years prior to 
Pathways involvement and still completed 2003 with higher scores on the standardized tests 
in both subject areas.” 

> “Pathways students experienced dramatically better school attendance, with participants 
attending 18.44 more school days and missing 9.57 fewer school days than their 
nonparticipant counterparts.” 

> “Middle school students in Pathways generally received better marks in English and math 
and more of the grades received by these students were satisfactory ones - a grade of C 
minus or better.” [ Final Evaluation Report, 21 st Century Community Learning Centers, 
Pathways to Progress, St. Paul, Minnesota, page 3.] 

Owensboro, Kentucky’s 21st Century Community Learning Center 

The Owensboro Public Schools’ 21st Century Community Learning Centers afterschool program 
serves students from a number of schools at five year-round centers. The program’s goal “is to 
provide safe, supervised, and fun learning opportunities for students in kindergarten through 
twelfth grade that serve to increase their academic skills, enhance their ability to interact 
positively with peers and adults, and reduce behavior problems.” [A Profile of the Evaluation of 
the 21st Century Community Learning Centers, Owensboro, Kentucky Public Schools, Harvard 
Family Research Project, 

http://www.gse.harvard.edu/hfrp/proiects/afterschool/rnott/owensboro.pdf, summarizing research 
by Robert J. Illback, Psy.D., and Benjamin Birkby, Psy.D. of R.E.A.C.H. of Louisville, Inc.] 

A three-year evaluation of the program, conducted by R.E.A.C.H. of Louisville, Inc., and 
released in three phases, evaluated a number of program implementation issues, as well as 
several student and family outcomes, including academic achievement and parental perceptions 
of the program’s ability to alleviate family stress. Among the findings: 

> “Seventy-seven percent of surveyed students reported that they were doing better at school 
because of the 21st CCLC program.” 

> “Parents continued to see academic benefits to their children due to the 2 1 st CCLC program, 
with surveyed parents reporting that their children look forward to school (88 percent) and do 
better in school (84 percent) because of the program.” 

> “At the six-month timeframe, approximately 50 percent to 60 percent of teachers reported 
that 2 1 st CCLC students had improved in turning in homework, completing homework, 
participating in class, regular attendance, attentiveness in class, academic performance, and 
preparedness for class.” 

> “Seventy-four percent of surveyed parents reported that the program helped their family have 
less stress.” 

> “Sixty-seven percent of surveyed parents reported that the program helped them remain 
employed. Twenty-four percent reported that the program helped them remain in school.” 
[HFRP Profile.] 

After School Education and Safety Program - Santa Ana, California 

With funding from California’s After School Education and Safety Program, Santa Ana, 
California in 1999 opened afterschool sites in four urban middle schools. The sites “serve 
predominantly Latino students with limited English proficiency and from high poverty 
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backgrounds. Although each site’s schedule varied, a typical program schedule included a one- 
hour homework period, a one-hour arts or life skills component, and a one-hour sports 
component.” [Harvard Family Research Project summary, 

http://www.gse.harvard.edu/hfrp/proiects/afterschool/mott/saaslsnpp.html, December 2003.] 

Jenel Prenovost, Ed.D., of the University of California, Irvine and the University of California, 
Los Angeles led an evaluation that relied on a quasi-experimental design, comparing the one- 
year results of three groups of students - a control group, a high-dosage treatment group 
(students who attended the program for 38 or more days during the school year), and a low- 
dosage group (students who attended for fewer than 38 days). 

The findings indicated that students in the high-dosage group showed better results than low- 
dosage and control group students. Results included the following: 

> “No statistically significant differences were found in SAT-9 Reading improvement scores or 
NPR [National Percentile Ranking] scores between treatment and comparison groups from 
1999 to 2000. However, high-dosage participants improved somewhat more than matches; 
this was especially true for eighth-graders, females, and students of limited English 
proficiency.” [HFRP at 

http://www.gse.harvard.edu/hfrp/proiects/afterschool/mott/saaslsnpp.html] 

> “No statistically significant differences were found in SAT-9 Math improvement scores from 
1999 to 2000 or NPR scores between treatment and comparison groups. However, high- 
dosage participants improved somewhat more than matches and all program participants 
improved more than the general school population. The subgroups of high-dosage sixth 
graders, high-dosage male participants, and high-dosage LEP [Limited English Proficiency] 
participants also improved more than matches. NPR scores favored high-dosage LEP and 
high-dosage LEP participants when compared to the low-dosage participants.” [HFRP] 

> “At two of the schools, there were significant differences in SAT-9 Math test scores between 
particular subgroups of program participants and comparison group matches. At the first, 
high-dosage males (p<.038) and high-dosage sixth graders (p<.048) improved significantly 
more that the matches. These same two subgroups also had significantly higher NPR scores 
on the SAT-9 Math test than matches. High-dosage participants at the second school had 
significantly (p<.084) higher NPR scores than low-dosage participants.” [HFRP] 

> “There were significantly (p<.005) fewer days of school missed by high-dosage participants 
(5.56 days) as compared to low-dosage participants (7.46 days) and the matches (6.80 days). 
In addition, high-dosage LEP students missed significantly (p<.002) less school than low- 
dosage participants and the matches. Higher-dosage sixth and eighth graders, on the other 
hand, had higher means in days absent than the matches, although this also was not 
statistically significant.” [HFRP] 

> “The program was associated with a nearly significant (p<.082) difference in improvement in 
school attendance. There was a statistically significant (p<.031) finding that high-dosage 
students improved more in their school attendance (1.36 days) than low dosage students (.32 
days). Also, high-dosage LEP students improved their attendance significantly (p<.05) more 
than low-dosage participants (.29 days) and matches (-.18 days).” [HFRP] 
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YS-CARE After School Program for California Work Opportunity and Responsibility to Kids 

The Los Angeles County Board of Supervisors established the YS-Care After School Program in 
1999. The afterschool program is “designed to offer a safe environment that includes academic 
assistance, homework help, enrichment activities, recreation, and quality childcare provided by 
caring adults in well-supervised school site environments.” [ Evaluation of the YS-CARE After 
School Program For California Work Opportunity and Responsibility to Kids (CalWORKS), 
March 2002, at http://www.gse.uci.edu/asp/aspeval/resourcesAfSCARE13.pdf, page 5, hereafter 
YS-CARE Evaluation.] The program is targeted at K-5 children attending schools in 
neighborhoods with high concentrations of families receiving Temporary Assistance to Needy 
Families (TANF). The Department of Education, University of California at Irvine and Research 
Support Services conducted a formal evaluation of the program, and released findings in March 
2002 . 

The evaluation compared the test scores and behavior of participating students with a 
comparable group of non-participating students. The study’s chief conclusions: 

> “YS-CARE participants had larger gains on SAT-9 Reading and SAT-9 Math scores than 
non-participants.” 

> “YS-CARE participants had larger gains on Reading Achievement than matched non- 
participants.” 

> “YS-CARE participants initially in the lowest decile reading group had significantly larger 
reading gains than matched non-participants.” 

> “YS-CARE participants had significantly lower scores on all Work and Study Habits and 
Citizenship measures at baseline. The participants narrowed the gap by the time of the end- 
of-year ratings, with almost half of the initial differences substantially smaller.” [YS-CARE 
Evaluation, pp. 5-6] 

Massachusetts After-School and Other Out-of-School Time Grant Programs 

In early 2003, the Massachusetts Department of Education released, via the Internet, a draft 
executive summary of an evaluation the state’s After-School and Other Out-of-School Time 
Grant Programs. The purpose of the program is “to establish or expand community learning 
centers that operate during out-of-school hours and provide students with academic enrichment 
opportunities along with other activities designed to complement the students’ regular academic 
program.” [Contact information at http://www.doe.mass.edu/contact/] The evaluation report was 
submitted jointly by Beth Miller and Wendy Surr of the National Institute on Out of School Time 
at Wellesley College, and Karyl Resnick and Kelly Church of School Enrichment Services of the 
Massachusetts Department of Education. The report covers Fiscal Year 2002. 

According to the report, “Building an Outcome Evaluation System For the Massachusetts 
Department of Education’s 21 st Century Community Learning Centers,” FY 2002 saw important 
gains for students in a number of areas. According to the report: 

> “Results indicate that 56 percent of the students participating had positive gains on measured 
outcomes.” [Building an Outcome Evaluation System .] 

> “Student gains in Math and/or English Language Arts were statistically significant in 73 
percent of the ASOST programs.” 
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> The areas with the greatest percentage of youth improving were Learning Skills, 
Communication Skills and Engagement in Learning. 

The Foundations After-School Enrichment Program 

For more than a decade, Foundations, Inc. has operated extended-day enrichment programs and 
provided technical assistance to other afterschool sponsors. During the 2001-2002 school year, 
Drs. Stephen P. Klein and Roger Bolus of Gansk & Associates, of Santa Monica, California, 
administered pre- and post-tests in mathematics and reading to first- through fifth-grade students 
in 19 Foundations programs in three states. A summary of the report is available on the 
Foundations website at http ://www.foundationsinc . org/ExtendedDayFolder/conclusions. asp . 

The full report, issued in December 2002, concludes: 

> “Foundations students made substantial improvements in average scores between the fall 
pretest and spring posttest. In fact, their average score gains in mathematics were somewhat 
greater than what would be expected given the results obtained in CTB/McGraw-Hill’s 
national norm sample. The Foundations students’ gains in reading kept pace with those 
made in this national norm sample.” [Improvements in Math and Reading Scores of Students 
who Did and Did Not Participate in the Foundations After School Enrichment Program 
During the 2001-2002 School Year, Stephen P. Klein, Ph.D. and Roger Bolus, Ph.D., Gansk 
& Associates, December 2, 2002, page 2.] 

> Over the course of the school year, Foundations’ afterschool students’ test averages moved 
them up national percentile rankings - by an average of 10 percentile rankings in 
mathematics reading, and an average of 2 percentile rankings in reading. Improvements in 
Math and Reading Scores, page 10.] 

> Foundations students fared very well by comparison to non-Foundations students at the 
studied schools. The mathematics “effect size” difference averaged .39 (representing 
39/1 00 ths of a standard deviation unit), in afterschool students’ favor. In reading, a similar 
finding: a .41 effect size advantage for afterschool students. [Improvements in Math and 
Reading Scores, page 14.] 

In September 2003, Drs. Klein and Bolus released a follow-up report, assessing of data from the 
2002-2003 school year, again finding statistically significant improvement. 

> “The major finding from this analysis is that there was a statistically significant improvement 
in scores between the pretest and posttest at every grade level and in every subject. 

Moreover, the amount of gain was greater than what would be expected given the results in 
the national norm group. For example, the average FOUNDATIONS’ student was at the 
40th percentile in mathematics on the pretest and at the 46th percentile on the posttest. The 
corresponding values for reading were 45th on the pretest and 48th on the posttest. Language 
arts was 43rd percentile on the pretest and 48th on the posttest. Results were similar within 
grade levels.” [Improvements in Basic Skills Scores of Students Who Did and Did Not 
Participate in the Foundations After School Enrichment Program during the 2002-2003 
School Year, page 7, Stephen P. Klein, Ph.D. and Roger Bolus, Ph.D., Gansk & Associates, 
Santa Monica, California, available from Foundations, Inc., by emailing 
info@foundationsinc . org . ] 
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> “We conducted regression analyses to assess the Foundations program’s overall effect on 
improving the students’ test scores. These analyses predicted a student’s posttest score on a 
test on the basis of that student’s pretest score on that same test, grade level, and “group” 

(i.e., Foundations versus non-Foundations). These analyses found that the students who 
participated in the Foundations program had about a 5-point higher posttest score (which is 
equivalent to about one tenth of a standard deviation unit) than did similarly situated non- 
participants (see Table 6). All the differences in Table 6 were statistically significant (at p < 
.10).” [ Improvements , 2002-03, page 11.] 

> The students who participated in the Foundations program during the 2002-2003 school year 
made substantial gains in math, reading, and language arts scores between the fall pretest and 
spring posttest. These gains generally exceeded the progress of students in CTB/McGraw- 
Hill’s national nonn sample. First graders in the Foundations program did particularly well, 
especially in mathematics. Foundations students also had statistically significantly greater 
gain scores between pretest and posttest than did comparable non- Foundations students who 
were tested under the same conditions and at the same time (see Table 6). It is not clear why 
fourth and especially fifth graders who were not in the program gained slightly (but not 
significantly) more between the pretest and posttest than did Foundations students. This 
anomaly may stem from a selection effect that was not accounted for by their pretest scores 
or perhaps to differences in how well the Terra Nova aligns with the Foundations’ curriculum 
across the different grade levels. Improvements, 2002-03, page IT] 

The Extended-Service Schools Initiative: 2002 Report 

In 1998, the Extended- Service Schools Initiative (ESS) began funding community organizations 
across the country to partner with local schools to create a total of 60 afterschool programs in 20 
communities. Each of the programs follows one of four nationally recognized program models - 
Beacon, Bridges to Success, Community Schools, or West Philadelphia Improvement 
Corporation - to provide youth-development activities in low-income areas in programs located 
in school buildings during non-school hours. ESS was funded as a five-year program by 
Wallace-Reader’s Digest Funds. Separately, the Wallace-Reader’s Digest Funds provided 
financial support to Public/Private Ventures (PPV) to conduct an evaluation of the program. 

PPV, with subcontractor Manpower Demonstration Research Corporation, began a four-year, 
multi-phase evaluation. Issued in June 2002, Multiple Choices: Findings from the Extended- 
Service Schools Initiative, is available online at http://www.ppv.org/content/reports/ess-multi- 
full.html . 

According to researchers, “Students who participated in the school-based, afterschool programs 
seemed to experience positive change in four key areas: staying out of trouble; improving their 
school attitudes and behavior; strengthening their social networks; and learning new skills, 
seeing new possibilities and improving their self-confidence.” [Multiple Choices, page 30, 
httpb/www.ppv.org/content/reports/ess-multi-full.html .l Specifically: 

> “Given that most of the ESS programs were new and the levels of participation were well 
below five days a week, it was not thought likely that we would observe changes in grades or 
test scores. However, to gauge whether ESS was starting to have positive academic effects, 
we asked parents and youth if they thought the program helped the youth do better in school. 
In addition, we measured some “leading indicators” of academic improvement (a sense of 
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academic mastery and the level of school effort) to ensure that we did not miss an important 
change if one had occurred. ... [Approximately two-thirds of the youth believed the program 
helped them do better in school, and it was even more likely that the parents found the 
program helpful to their children in this way.” [Multiple Choices, page 32.] 

> “Interestingly, the parents’ survey responses are consistent with the expected pathway of 
change that could ultimately lead to increased academic success. High percentages of parents 
felt that ESS helped their children like school more and try harder in school, factors that may 
lead to learning more and doing better.” [Multiple Choices, page 32.] 

> “When we examined how the youth’s academic attitudes and behaviors changed over time, 
we found a consistent story. Youth who participated in ESS activities experienced a greater 
increase in their sense of belonging at school and paid more attention in class. Again, 
consider the two groups of similar youth. . . . [A]mong the youth who did not go to ESS 
during the 13 months between the initial and follow-up surveys, 20 out of 100 reported that 
they started skipping school, 29 said they really paid attention in class, and 76 said they were 
very proud to belong to their school. Among similar youth who went to ESS two days a 
week, only 1 1 out of 100 reported starting to skip school; 49 said they really paid attention in 
class; and 84 said they were very proud to belong to their school.” [Multiple Choices, page 
32. | 

> “Responses on the parent survey administered in Spring 2001 suggest that the after-school 
programs were having some of these beneficial outcomes: 80 percent of parents said they 
were less worried about their child’s safety after school. 57 percent said their child’s 
participation helped them manage their own work schedule. 47 percent said it let them attend 
classes or job training more easily. 45 percent said it helped them get a better job or do better 
at their job.” [Multiple Choices, pages 33-34.] 

North Carolina’s ‘Support Our Students’ 

In 1994, the state of North Carolina launched its “Support Our Students” initiative (SOS), to 
provide funding for afterschool programs across the state. The program offers grants in the 
$60,000 to $250,000 range to nonprofit organizations in the state - one per county, each of 
which coordinates services in their counties. In 2001-2002, the program provided $12.5 million 
to nonprofits in 98 counties. In all, the program supported programs in 190 school-based sites, 
and 54 community-based sites, providing afterschool services to 16,000 students during the 
school year, and summer programming for 10,000 students. 

An evaluation of the 2001-2002 year’s programs, conducted by EDSTAR, an independent 
research and analysis firm based in Raleigh, North Carolina, found the following: 

> “At every grade except sixth grade, improvements in SOS participants’ mean EOG [End of 
Grade Achievement Test] reading scale scores exceeded the state’s improvement goals.” 
[North Carolina Support Our Students 2001-2002 Program Highlights, at 
http://www.edstar.org/sos 2002reports/021009 SOS handout.doc] 

> “Classroom teachers reported that more than 40 percent of the regularly attending 
participants improved their grade in English and/or math.” 

> “The percentages of students who scored at grade level proficiency increased in both reading 
and math, with the greatest increase in reading — from 67 percent at grade level to 7 1 
percent.” 
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> “Minority students made greater improvements than White students in both math and 
reading.” 

> “African Americans made greater improvements in reading than any other demographic 
group; minority participants overall made nearly twice the gain of White students in reading 
scores.” 

> “Students continued to make steady, consistent improvement each year they participated in 
SOS. The average yearly improvement was slightly less than half a proficiency level.” 

> “Of the three-year SOS participants, more than two thirds had improved at least two 
proficiency levels in reading and math, compared with the year before joining SOS.” 

> “Except for sixth graders, SOS participants who were the furthest behind and had the most 
risk factors (e.g., free/reduced lunch status, single-parent households, etc.) made the greatest 
gains on EOG. Evaluators surmised that sixth graders often had problems making the 
transition to middle school, and recommended that sixth-grade transition programs be 
implemented.” 



* * 

The Afterschool Alliance is a nonprofit public awareness and advocacy organization supported 
by a group of public, private, and nonprofit entities dedicated to ensuring that all children and 
youth have access to afterschool programs by 2010. The Alliance is proud to count among its 
founding partners the C.S. Mott Foundation, U.S. Department of Education, JCPenney 
Afterschool, Open Society Institute/The After-School Corporation, the Entertainment Industry 
Foundation and the Creative Artists Agency Foundation. Washington, DC Office: 202/347- 
2030; Flint, Michigan Office: 810/239-3449 
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